*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task

We present *-CFQ ("star-CFQ"): a suite of large-scale datasets varying scope based on the CFQ semantic parsing benchmark, designed for principled investigation scalability machine learning systems in realistic compositional task setting. Using this suite, we conduct series experiments investigating ability Transformers to benefit from increased training data size under conditions fixed computational cost. show that generalization remains challenge at all sizes, and increasing natural language leads consistently higher error rates, which are only partially offset by data. further while additional related domain improves accuracy data-starved situations, improvement is limited diminishes as distance target increases.